Phase B3b — zero-copy persisted FM-index - #16
Merged
Conversation
Capability-anchored (build-once -> instant-start, bounded, reproducible). BwtBacking seam (one algorithm, owned + borrowed backings); deterministic 8-aligned LE on-disk format extending genomics/index/format.rs; dependency-free zero-copy via align_to; faithful serialization in v1 (lean-rank is a versioned v2); equivalence + determinism + no-rebuild gates. Decomposed B3b.1 (trait) -> B3b.2 (format+view+gate+docs/index-format.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8-task plan across two parts: B3b.1 (BwtBacking trait + generic FM ops, route BlockedFMIndex through it, behavior unchanged) and B3b.2 (8-aligned LE on-disk format + deterministic GenomeIndex serializer + zero-copy FmIndexView/GenomeIndexView + view==in-RAM equivalence/determinism/no-rebuild/bounded-residency gates + docs/index-format.md). Derived from the committed spec (4c55b8a) and grounded in the existing fm_index/rank_select/sampled_sa/index code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracts the data surface the blocked FM query algorithm reads into a pub(crate) BwtBacking trait, and rewrites rank/symbol_at/lf_index/backward_search/sa_at/locate_interval/total as free functions generic over it. The single source of truth for the algorithm; the owned BlockedFMIndex (next) and the persisted FmIndexView (B3b.2) both run it, so the borrowed view can only diverge on backing, not logic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BlockedFMIndex implements BwtBacking over its owned fields; backward_search/locate_interval/rank/total/symbol_at/sa_at become thin wrappers over the generic fm_backing ops, and the private lf_index is removed (now fm_backing::lf_index). No observable behavior change — the FM-index/aligner/genome_index suites and tests/fm_index_props are the witnesses. The inherent c_table() -> &[u32;6] is kept for align_within_block; the trait's by-value c_table coexists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…itives
Adds additive pub(crate) accessors so the B3b serializer can read each structure's backing words (BWTBlock::{start,end,bwt,occ,sentinel_offset}, RankSelectIndex::{bitvectors,superblocks,len}, SampledSuffixArray::{marks,superblocks,values}, CompressedBoundaries::len), and promotes popcount_range + RANK_STRIDE to pub(crate) so the borrowed FmIndexView computes rank with the same code as the owned structures. No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ions Adds Reference2bit/FmMeta/Boundaries/Blocks section kinds (the B3b on-disk format extends the existing versioned v1 header; Reference stays reserved). The fixed header is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
IndexWriter::write_genome_index serialises a built GenomeIndex into the 8-aligned little-endian section format (Contigs+global_offset, Reference2bit, FmMeta, Boundaries, self-describing Blocks directory+records, SaSamples). Byte-identical across repeated builds. The scaffold write_v1/ContigInfo and the old reader are removed; IndexReader::open is stubbed pending Task 6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
open() mmaps the file, validates the header + every section extent/8-alignment, parses the new Contigs layout into a ContigSet (recomputing and integrity-checking global offsets), and locates the FM sections. Rejects truncated/bad-magic files. The borrowed FmIndexView (Task 7) interprets the FM sections. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…equivalence gate FmIndexView reads the FM sections as &[u64]/&[u32] over the mmap (checked align_to, little-endian host) and implements BwtBacking, so backward_search/sa_at/locate_interval run the same generic ops as the in-RAM index. GenomeIndexView pairs it with the ContigSet for locate_exact. The equivalence gate asserts byte-identical backward_search/sa_at/locate_exact/rank/symbol_at over a multi-contig + N-bearing battery. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per code review: FmIndexView::new now validates every block record (directory offset + declared word-counts) fits within the Blocks section using checked arithmetic, so a corrupt-but-open-valid file is rejected at open() rather than panicking or silently wrapping into a wrong-length slice at query time (release builds have no overflow checks). Also asserts the SaSamples rate equals the FmMeta sa_sample_rate (removing the previously-unused SampledView.rate field). Adds open_rejects_a_corrupt_block_directory. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Integration gates over the public API: view == in-RAM locate_exact over a multi-contig + N battery, byte-identical builds (two independent builds), self-contained query (in-RAM index dropped), and bounded residency (FmIndexView is a small borrow, not an owned copy). Documents the versioned on-disk format as a forkable contract incl. integrity validation and the v2 lean-rank path. Removes the unused RankSelectIndex::len accessor for a clean build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final-review fix: validate_block_records now rejects a block-record directory offset that is in-bounds but not 8-aligned, so a corrupt index is rejected at open() rather than panicking at query time via block()'s align_to .expect (the integrity contract: corrupt is rejected up front, never crashed-on later). Adds open_rejects_a_misaligned_block_offset. Also: drop a useless vec! in the header reserve, and document the MSRV-mandated (n+7)/8*8 (not div_ceil) at the two block-size sites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The keystone of Phase B's persistence pillar: build the reference index once into a portable artifact; thereafter every analysis memory-maps it (never rebuilds), runs in bounded memory, and is byte-identically reproducible — delivered at the library level.
GenomeIndexto a deterministic, versioned, 8-aligned little-endian on-disk artifact (IndexWriter::write_genome_index) and loads it as a zero-copy, memory-mappedFmIndexViewthat answersbackward_search/sa_at/locate_exactin place — no rebuild, no allocation of the bulk index.pub(crate)BwtBackingtrait (src/genomics/fm_backing.rs); both the in-RAMBlockedFMIndexand the borrowedFmIndexViewimplement it, so the view cannot diverge on logic — only on how it fetches bytes. A byte-identical equivalence gate proves it.IndexReader::openfully validates the artifact: magic/version/endian, every section's extent + 8-alignment, contig global-offset consistency, per-block-record extents + 8-alignment, and SA-sample-rate consistency. Corrupt/truncated/misaligned files are rejected at open — never mis-queried or crashed-on at query time.slice::align_to(little-endian host; nobytemuck).docs/index-format.mddocuments the format as a forkable contract, including the faithful-serialization size note and the v2 lean-rank path.Scope: library-level (B3b). Deferred by design:
rosalind indexCLI + load wiring (B3c); wiring align/variants/pileup onto the persisted index (B4);MemoryBudgetenforcement (Phase C); leaner/faster rank (format v2).Spec gates — all met
N-bearing + boundary-straddle battery (exhaustiverank/symbol_atover every position, plusbackward_search/sa_at/locate_exact).BlockedFMIndex::build(structurally grepped + behaviorally proven by a self-contained query after dropping the in-RAM index).FmIndexViewis a small borrow (slices + scalars), not an owned copy; size independent of genome size.BwtBackingrefactor changes no observable behavior (existing FM-index/aligner/genome_indexsuites +fm_index_propsare the witnesses).Test plan
cargo test— full suite green (23 test binaries, 0 failures)cargo build— 0 warningscargo fmt --all -- --check— cleantests/index_persistence.rs(public-API equivalence battery, determinism, self-contained query, bounded residency) + theview == in-RAMequivalence gate + corruption-rejection unit tests (truncated / bad-magic / corrupt block directory / misaligned block offset)div_ceil); twounsafe align_toblocks reviewed and found sound (checked prefix/suffix + LE-host guard + per-record validation at open)Design spec:
docs/superpowers/specs/2026-05-27-phase-b3b-zerocopy-index-design.md. Plan:docs/superpowers/plans/2026-05-27-phase-b3b-zerocopy-index.md.🤖 Generated with Claude Code